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ABSTRACT 

This report summarizes one of a series of workshops 
organized by RAND' s Critical Technologies Institute (CTI) , on behalf 
of the U.S. Department of Education, to learn from those already 
implementing new technologies in schools. This particular session 
explores the educational software market from the demand side and 
focuses on issues of cost effectiveness and evaluation. Five case 
studies of school-wide technology implementation and its benefits are 
presented, along with a chart comparing the costs incurred by the 
five schools to each other and to the national average. Synopses are 
provided for four presentations sampling the instructional 
effectiveness of educational software. The summary addresses issues 
of implementation such as teacher preparation, financing, rearranging 
the schoo’ day to allow computer time, focusing the technology into 
courr eware and individualized instruction, and monitoring student 
progress. Participants also advise on prioritizing for successful 
technology plans and for choosing instruments with which to evaluate 
software tools. Appendices offer a workshop schedule and a list of 
participants . (BEW) 
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Urged by the President and the Vir:e President, federal officials 
have been exploring how to encourage greater and more effective use of 
modem computers euid communications in th»5 nation's schools. In July 
1994, RAND's Critical Technologies Institute (CTI) completed a broad 
investigation of educational technology in education eind training for 
the White House Office of Science and Technology Policy and the National 
Science and Technology Council. This work examined the nature and level 
of existing federal efforts. 

Based on this work, the U. S. Department of Education asked CTI to 
assist the Department as it responded to new provisions in the 1994 
GOALS 2000: Educate America Act, provisions calling on the Secretary of 
Education to develop a strategy for effective utilization of the new 
technologies in the nation's classrooms. With an original deadline of 
March 1995, submission of the plan to the White House and the U. S. 
Congress was delayed by later legislation until September 1995. 

This report summarizes the fourth of five workshops organized to 
learn from those already involved in implementing use of the new 
technologies in the schools. The first examined the educational software 
market from the supply side; the second issues of professional 
development; the third local planning and financing; end the fourth the 
educational software market from the demand side. This workshop was 
primarily concerned with issues of cost, effectiveness and evaluation. 
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1. TECHNOXjOGY, school REFOKM'S OPPORTDKlOry 



1MTRODT7CTZON 

The last in a series of workshops held over a period of 18 months, 
this two-day workshop on J^me 1-2, 1995, was organized primarily to 
gather infoirmation and expert opinion on the potential benefits and 
effectiveness of the school use of computer technology. Over the course 
of two days, workshop participants heard and considered 14 
presentations, including summary reviews of the implementation and 
benefits of the school -wide use of computer technology in five 
pioneering technology-rich schools; four reviews of experimental and 
empirical data on the use of computer technology in order to implement a 
familiar curriculum component or realize a well-defined goal in K-12 and 
adult education, and military training; a prepared paper on the 
technology related cost^ for technology-rich schools; and a prepared 
paper discussing factors affecting what can be learned about technology 
in education from traditional evaluation methodology'. 

Discussion among workshop participants covered the important issues 
of ... 

• benefits: importa*it educational outcomes attributable to using 
information technology, e.g., preparing students for the world 
of work; 

• costs: front-end and recurring, for hardware, software, teacher 
training, curriculum preparation, technical support staff, 

etc . ; 

• effectiveness; teaching and learning familiar school coursework 
better, e.g., elementary arithmetic skills, using technology; 

• implementation: planning, buy-in by teachers, and financing; 
and, 

^There are costs for equipment, materials, infrastructure !e.g., 
cabling), and technolocfy related training. The costs may be support.*;d 
either by addition of new resources or substitution for existing 
activities . 
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• evaluation: what's different about technology. 

No effort was made to reach any formal workshop conclusions, but a 
consensus readily emerged among workshop participants on the 
t echno 1 og i ca 1 oppor t uni ty that ex i s t s to improve s t uden t 1 eami ng and 
school effectiveness at a significant but acceptable price. In addition 
to the prima facie benefits of motivating students to higher levels of 
educational attainment and preparing them for the world of work, 
computer technology (with suitable software) can: 

• guide students to the improved acquisition of certain basic and 
advanced skills and 3cnowledge; 

• aid students in the conduct of projects, including projects 
that may require resources outside of and even remote from 
school, which lead to the acquisition of other advanced skills 
and knowledge ; and , 

• assist the student and teacher to track and maintain a record 
of the student's learning progress and performance. 

A copy of the workshop agenda appears in Appendix 1; a list of 
p'articipants in Appendix 2 . A summary of the workshop proceedings 
follows . 

scbooZj-wxde technology xkplementatxons and thexr benefits 

We can make no pretense that these school data (below) are 
representative of U.S. schools. Three criteria governed the select-ion 
of schools: (1) that the schools were technology-rich outlyers, along 
different dimensions; (2) that they willingly cooperate with Rand^ in 
assembling relevant cost data; and (3) that the technology 
implementation^ was sufficiently long-lived and stable so that at least 

^ B. Keltner R. and Ross, The Cost of High Technology Schools, 
Critical Technologies Institute, RAND, forthcoming. 

^The phrase technology implementation is not intended to mean 
hardware alone, but includes other necessary components like software, 
teacher training, curriculum preparation, technical support staff, 
building cabling or rewiring, and so on, but merely substitutes for this 
clumsy locution. A technology implementation may may not be a part 
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initial evidence of benefits were visible and could be cited. 
Representatives of five schools, elementary, middle or junior, and 
secondary schools meeting these criteria were selected to participate in 
the workshop. No doubt other schools could have served as well. The 
number was limited by the practical consideration of leaving sufficient 
time ?n a two-day workshop for discussion among participants. The 
siammaries below aim to give the reader a sense of the variety subsumed 
in the phrase "technology-rich* school. 

Blackstock Jr* E*S* 

Steve Carr, a teacher, described the technology implementation at 
Blackstock Jr. H. S. , a 6^^ through 8^^ grade school in Port Heuneme, 
California. The description (below) draws on Carr's oral presentation as 
well as excerpts from the Keltner and Ross paper. (Excerpts from the 
paper appear below in italics.^) With annual per student expenditures in 
1994 of $4,060^ for some 960 students, many eligible for Title 1^ 
support, this 36-classroom school caters to a largely minority 
population of mostly Hispanic descent, with smaller nxjmbers of Chinese 
and Vietnamese students . Twenty-two percent of the student body are 
characterized as having limited English-language skills. 

''Blackstock 's model of educational technology delivery centers on 
creating what are called 'smart classrooms , '* There are at present eight 
smart classrooms, including two for instruction in 7^^ grade science, 
one for instruction in 8^^ grade science, two for literature and 
history, one for ESL instruction, one for instruction in business 
education, and o,r;e called the Tech Lab 2000.^ Each has been conceived 



of a school reform effort that aims at new student outcomes, new 
approaches for assessing student outcomes, and new instructional 
strategies . 

^ Excerpts from the Keltner and Ross paper are used extensively 
throughout the Current Evidence section to supplement oral presentations 
by school representatives. They appear in italics, within quotation 
marks. They are taken from the draft of the paper and may differ 
slightly from their final report. 

^ This compares with a national average of under $6,000 in 1994. 

® Title 1 of the Elementary Secondary Education Act of 1965 
(reauthorized in 1994) . 

A mathematics smart classroom, nearly completed, will bring the 
total to nine. 
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and designed to support a technologically-intensive educational 
delivery, 

"The Tech Lab 2000 is perhaps most appropriately described as the 
futuristic equivalent of a wood or metal shop. Designed to make students 
familiar with the technology present in the modem workplace, the Tech 
Lab is outfitted with Computer Assisted Design (CAD) software, a 
Computer Numerically Controlled (CNC) flexible manufacturing system, 
pneumatic equipment, and a satellite dish. All of the other smart 
classrooms have between 25-30 coirputers on a local area network (LAN) . 
Each is also equipped with a sophisticated file server and a SOTA switch 
to give the teacher maximum control over classroom dynamics. The switch 
makes it possible for the teacher to control which software programs are 
running on each individual computer. Students can all be working on the 
same project, e.g., a software program or an interactive video 
presentation, or there can be as many as 12 to 15 different activities 
going on in the classroom at the same time. 

''There is also plenty of technology outside of the classrooxns . In 
each of the schools' other classrooms, there are banks of ten computers 
and two printers . Teachers in the non-smart classrooms do not have the 
same sophisticated management system to control technology delivery, but 
are able to use many of the basic and important software applications , 
from word processing to interactive programs , in their instruction. They 
can also draw on the school's connection to the Internet to create a 
more technologically rich environment. 

"Staff development efforts for teachers in the smart classrooms 
have centered on giving individual instructors large amounts of paid 
tijne-off to familiarize themselves with technology and to organize a 
technology-based curriculum. Of the eight teachers in the smart 
classrooms, four took a year off and one took two years off to prepare 
themselves. The other three teachers were given three weeks during the 
summer to prepare. In the latter cases, the teachers were setting up a 
second smart class in a subject area where one already existed. The 
presence of a teacher with technological and curricular know-how made it 
easier for the new teacher to get up and running more quickly. Ongoing 
staff development for all teachers , those in smart and non-smart 
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classrooms alike , is supported by four paid days of technology training 
per year and a considerable amount of informal networking. 

"Up to the present, Blackstock has not had a technology coordinator 
to support staff development efforts, relying instead on paid leave time 
and informal networking. To keep the technology program ruxming 
smoothly, there is a teacher who has devoted about a quarter of his time 
to technology-related problem-solving and to computer repairs. Starting 
next year this teacher^ will move into the position of full-time 
technology coordinator. ^ 

Noteworthy points arising in Carr's presentation include: 

• The technology program at Blackstock is now over eight years 
old, and the issue of recurring costs for hardware, software 
and even special furniture, often ignored in the excitement 
attendant on program start-up, was demanding increasing 
attention. 

• Program start-up was only possible with a $2 . 5 million grant 
(over five years) from the California Model Technology Schools 
Program. 

• Classroom furniture and organization are important 
considerations for teachers responsible for smart classroom 
design, and changed over time with experience. 

• The technology program was implemented in an established and 
largely unchanged curriculum framework. 

• Continuing staff development depends upon cooperation ctmong the 
school's teachers, as opposed to district level efforts. 

• Important outcomes of the technology implementation are 
improved student attitude and engagement, based on livelier 
classroom content, and improved student achievement, measured 
by test results. 

® This is Steve Carr, a history teacher, who represented Blackstock 
at the workshop . 
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Christopher Colusbus Middle School 

Fred Carrigg, dii^^^tor for academic progpraios of the Union City 
School District, and Bob Fazio, principal of the Christopher Columbus 
Middle School, described the educational context and technology 
implementation at Christopher Columbus (CC) , a small 7^ and 8^^^ grade 
school of 310 students in Union City, NJ. Reflecting the school 
district's student population, the largest niomber of CC's students are 
Hispanic. Many to most do not speak English at home, are enrolled in the 
ESL program, and are eligible for free or reduced cost Ivmch in school. 

A 'whole langtiage philosophy of education', a project-based rather 
than textbook-based approach to curriculum and instruction, and a 
reorganization of the school day into a smaller number of larger time 
blocks are the basis for CC's technology implementation. From Keltner 
and Ross: ^'Technology has been used to create a 'research-based' 
curriculum. The school's curriculum integrates traditional subject 
areas, but has as it's main focus an emphasis on teaching students 'how 
to learn. ' Students are encouraged to become active learners through the 
use of structures research activities and group project work. To 
facilitate the transition to a student -centered learning environment, 
instructional delivery at the school^ has been reorganized. Rather than 
the traditional 50-minute period, classes meet for between one-and-one- 
half hours and two hours. The longer class periods allow students to 
delve deeper into their coursework and give teachers more time to act as 
educational facilitators . 

"Each of the school's twelve classrooms is outfitted with five 
computers (a mxx of Macs and PCs ) , a printer , and a video presentation 
station (VCR, laserdisk player and presentation monitors) . There are 30 
additional Macintosh computers with CD-ROM capabilities in the school's 
central computer lab. To allow students to experiment with multimedia 

^ Bell Atlantic has worked with the Christopher Columbus Middle 
School over the past two years to add a high-speed school -and-home 
computer-communications network to the school technology program. The 
network involves the use of high-speed telephone lines (ISDN) to connect 
school computers and 150 student and teacher homes to a library of CD- 
ROM and software titles stored centrally on six file servers at a Bell 
Atlantic site. This component of the CC technology program 
experimental and is not further described here. 
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production, the cowputer laJb is also outfitted with cstmcorders, a video 
projector, and a coiiputrer video editing unit. The school has two LANs, 
one for Macs, the other for PCs. The PCs are linked to the Internet to 
allow remote resources to be integrated into classroom instruction. 

"To ge: CC's technology program up and rxmning, each of the 
school 's 15 teachers were given six days training in each of the first 
two years of implementation. After the two-year start-up period, staff 
development continued at a lower level of intensity , with each teacher 
receiving an average of three days of paid on-going training per year. 

To keep the school 's technology program running smoothly, there is a 
full-time technology coordinator on-site. The technology coordinator is 
responsible for conducting student computer classes, supporting 
teachers, and making technology repairs.'' 

Noteworthy points arising in the Carrigg/Fazio presentation 
include : 

• External corporate financial support was critical to successful 
initiation of the technology implementation in order not to 
stimulate a competition among existing programs for traditional 
budget support . 

• The school day has now been restructured into uninterruptible 
148 minute blocks of time for whole language instruction that 
combines English and social studies; and 111 minute blocks of 
time combining mathematics and science education. 

• Classroom practice emphasizes student-centered, small-group, 
project-based learning. 

• Planning aims to expand the computer presence in classrooms, as 
opposed to the computer lab. 

• Important outcomes of the technology implementation are rising 
attendance and test scores on normed stauidardized tests, now 
already above the NJ average (although below the national 
average) and reduced Title 1 eligibility. 
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E. Bak^rsfiald High School 

John Gibson, principal of East Bakersfield High School in 
Bakersfield, CA, described the technology-rich, school-to-work 
transition program of this 2400 student high school, with a majority 
Latino population and an educational philosophy that education equals 
experience. From Keltner and Ross: "The school's chief administrator 
aims to have students understand early that their high school education 
shapes their job prospects, and that their present educational 
experience is a way of building job-relevant skills. Elxposure to 
business and career-oriented themes begins immediately in the ninth 
grade and continues throughout their high school education, and includes 
resume writing, portfolio building and project activities oriented 
towards the local business community. 

"The school's curriculum is organized around five career 'tracks'. 
The career tracks are not targeted at specific ability levels, nor do 
they consist of a core set of classes that each pupil must complete. 
Rather, they are designed to allow students to develop technical and 
applied skills related to broad industry groups. One career track is 
oriented around coursework in science, technology, engineering and 
manufacturing (STEM) . Included in this curriculum is everything from a 
freshman class in the principles of technology to advance placement 
physics for seniors. Students in this career track can make use of the 
Hands on Science & Technology (HOST) Center to use technology in the 
design and fabrication of exhibits . A second career track prepares 
students for employment in health-care. The school's health careers 
academy has 200 professional partners throughout the Bakersfield area, 
which offer students internships during the school year and the summer 
break. A third career track is Communications and Graphic Arts, in which 
students have the opportunity to use video cameras, video toasters and a 
computer editing device. . 

Another career track is known as human and government services, 
designed to prepare students for careers in teaching, law and public 
administration. Particular attention is given to developing strong 
skills in both written and oral communication. The remaining career 
track is oriented towards developing business and entrepreneurial 
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skills. Students can participate in a one-semester class called EB 
enterprises, in which they carry out projects in a high-tech office 
environment for teachers, school administrators or community businesses. 
Project work includes developing inventory programs, generating 
descriptions of courses and scholarships, and doing graphics for signs 
and brochures. Students alternate as office managers in order to learn 
how to manage tasks and coworkers . 

"Technology-based instruction is integrated smoothly into 
coursework from beginning to end. As freshmen, students take a nine-week 
course in keyboarding and basic computer literacy. Writing assignments 
in the freshman English and history core courses are organized to ensure 
that all students moving into their sophomore year are proficient in the 
use of word processing programs. As seniors, students have to coit^lete a 
techno logy- based project as a graduation requirement . Projects involve 
the use of computers, graphics software or video equipment. 

"General instruction between the first and final years is heavily 
technology- based. Math classes integrate an interactive math program. 
English, history and social studies teachers have access to writing labs 
as well as a large number of video towers equipped with CD-ROM, 
videodisc players and VCRs. The school building is in the process of 
being rewired to accommodate network technology. Next year, many of the 
classrooms will have Internet connectivity. 

"Administrators at E, Bakersfield use a variety of measures to 
support technology-related staff development. There is a limited amount 
of funding available for paid, formal technology training — the school 's 
staff development budget allocates an average of one paid day per 
teacher per year. Much of this budget goes to training new teachers. New 
teachers without any prior training in computer technology are expected 
to spend several days during the summer break in training to achieve 
basic fluency. New teachers with more experience are typically requested 
to train on their own time. To support informal development efforts, the 
school has a teacher lab equipped with nine computers and a laser 
printer. Many of the computers have CD-ROM capabilities. To keep the 
technology component of the school running smoothly, the school also has 
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a half-time technology coordinator, a full-time repair specialist and a 
budget for hiring network specialists on an as needed basis. ^ 

Noteworthy points arising in Gibson's presentation include: 

• Multiple sources of external funding, including the California 
Model Technology Schools Program were critical to successful 
development of the school's technology program. Meeting 
recurring costs is a growing, serious problem. 

• In-situ assessment of the student's performance is seen as a 
major need. Students already maintain an individual projects 
portfolio on diskette . 

• Important outcomes of the technology program are improved 
student retention and irrproved job placement of graduates. 

Northbrook Middle School 

Susan Wolf, principal of the Northbrook Middle School in Houston, 
TX, described the school's technolog^i^ implementation, initiated in an 
existing building with a $6 million start-up budget in 1991. This 6^^ 
through 8^^ grade school of xinder 800 students draws its students from a 
diverse population of ethnic families of which about 70 percent are 
Latino . 

From Keltner and Ross: ”The school administrators understand their 
main mission to be the preparation of their studencs as life-long 
learners for the world of work. The school's curriculum, while centered 
on traditional academic subjects, places heavy emphasis on students 
acquiring critical thinking and problem solving skills. Teachers are 
expected to assist students in learning how to find and analyze 
information. To support this student-centered learning environment, the 
school is organized into four educational clusters. Teachers and 
students in each cluster work together to support one another in 
continually expanding their ability to g^^ther information and solve 
problems. Technology is viewed as a primary vehicle to help students 
develop critical thinking and problem solving skills. Technology permits 
instruction to be tailored to individual student needs. 
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''Northbrook's technology program is centered primarily on the use 
of computers. With over 400 conputers in place in the school's six 
technology labs and 48 classrooms, Northbrook has a student to conputer 
ratio of just under 2:1. Each of the school's classrooitis is outfitted 
with between five and six computers. All of the computers have access to 
CD“ROM drives in order to expand the range of software products 
available for student use. Access to network resources are used to 
support student information searches. Computers in the classrooms, in 
the computer labs, and in the library are networked together in a 
school -wide LAN with Internet connectivity. Video technology allows 
viewing on three channels, including local origination. Teachers also 
make use of multimedia presentation equipment. Each of the classrooms is 
outfitted with a videodisc player, videocassette recorder, television 
receiver, and LCD-equipped overhead projector. 

"To support the technology program, Northbrook has relied primarily 
on on-site staff development. Each of the school's 48 teachers received 
two weeks of technology-related staff development in the summer prior to 
the school start-up . On an ongoing basis, teachers participate in, on 
the average, three to four days of paid training each year. Additional 
personnel to support the technology program include a full-time 
technology facilitator and a district technology coordinator located on- 
site. These two individuals conduct in-house training and keep the 
technology program ruxming smoothly. 

Noteworthy points arising in Wolf's presentation include: 

• Cf the $6 million start-up budget, $1.5 million was allocated 
for technology. Recurring costs are a major concern. Title 1 
funding used for this purpose does not adequately address the 
problem. 

• Significant technology applications are drill-and-practice and 
productivity use. 

• Classes have been extended to 90 minutes to allow for 
curriculum integration, e.g. , multidisciplinary connections and 
collaborative project work mediated by computer. 
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• Sharply improved test scores cannot be attributed to technology 
alone. Other factors are excellent teachers, staff development 
cuid flexible scheduling for student-centered approach to 
learning . 

Taylorsville Elaaumtazy School 

Beth Stroh, Modem Red Schoolhouse^^ project coordinator at 
Taylorsville Elementary School described the technology in^lementation 
of this suburban pre-K through 6th grade school in Taylorsville, IN with 
its largely lower middle-class, white student population numbering 615. 

From Keltner and Ross: ^Taylorsville is one of several schools in 
Indiana working with the Modem Red School House (MRSH) educational 
design team - a New American School Development Corp. WASDC) activity - 
to bring information technology into its educational delivery . The 
school's technology plan, its hardware layout, and its staff development 
effort reflect the essentials of the MRSH design. The most iirportant 
role for technology in the school 's educational design is to support a 
commitment to self-paced individualized learning , 

Taylorsville's curriculum enphasizes core subjects, aiming for high 
levels of proficiency in language arts, math, science, history and 
geography. Despite this emphasis on standardization in content, 
educational delivery focuses on students proceeding through coursework 
at their own pace. Instructional strategies promote multi-age, multi- 
year groupings and stress team-based project work. The opportunities for 
regrouping teams during project work allows individual students to 
develop their skills in different areas at an appropriate speed. By 
virtue of their role in integrating instruction across subjects and 
grades, teachers play a key role in facilitating the transition to a 
self-paced student environment. 

The school's technology plan provides students with plentiful 
access to networked computers. Taylorsville has one computer lab 
equipped with 30 Apple computers. Each of the school's 25 classrooms has 
a cluster of four student computers, one teacher computer, and two 

A New American Schools Development Corp. (NASDC) project design 
activi ty . 
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printers • Some of the classroom computers have internal CD*‘ROM drives to 
increase the range of software applications accessible to students . A 
school-wide LAN connects classroom computers to the computer lab and to 
administrative offices. At present, students can access the Internet 
from two computers in the library media center. Plans provide for 
Internet connectivity to each classroom. Investing in the hardware and 
other infrastructure required to give each classroom Internet 
connectivity is an outcome of the school's commitment to supporting 
student project activity. The same principle has led also to outfitting 
the library with eight IBM clones that use sophisticated software to 
facilitate information and reference searches. 

To support its vision of a self -motivated, self -directed student 
population, the school invests in a fairly high level of staff 
development. In Taylorsville's educational paradigm, teachers serve as 
facilitators for student learning. Teacher fluency and comfort in using 
information technology determines the success of the model. In the first 
two years of implementation, staff received six full days of technology 
training per year. Thereafter, two days a year have been devoted 
specifically to ongoing training in technology. A full-time technology 
coordinator asr.irts teachers with their technology- related problem 
solving. The full-time technology coordinator has the assistance of 
three part-time aides. " 

Noteworthy points arising in Stroh's presentation include; 

• Once committed to the MRSH design, teachers recognized 
technology as an important enabling tool for developing 
education compacts and instructional strategies adapted to 
individual learners; and for continuously measuring student 
progress in new on-line formats, including student portfolios, 
and recording student scores on standard assessment 
instruments . 

• Although initially discovered to be in short supply, an 
exceptionally capable instructional management software (IMS) 
product was finally acquired at substantial cost that provided 
for excellent linking between curriculum elements, maintained 



individual education compacts, and provided features for on- 
line testing. 

• Students successfully engaged in independent, self-paced work, 
but still challenging problems remain for teachers to manage 
individualized student learning and be confident that students 
are achieving at their highest level . Assessment of student 
project work remains a problem. 

• Important outcomes of the MRSH implementation include increased 
skills and confidence with technology, increased student 
enthusiasm for learning, increased commitment to being 
responsible for own learning, and some iitqprovement in 
standardized test scores, most consistently as a result of 
improved reading skill. 

COSTS 

Table 1, taken from Keltner and Ross, shows the cost of technology 
implementations for pioneering schools like these. In addition to data 
on these five schools, Table 1 includes data on three school -wide 
designs sponsored by NASDC.^^ 



As Rand is the principal planning and evaluation contractor for 
NAS DC , these data were conveniently available. 



COHPAHATIVE COST FIOUHES 
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The cost figures shown are not computed from actual historical 
prices, but rather as the amortized cost of the school configuration 
based on today's prices. Per Keltner and Ross, the following rules and 
assumptions were used: 

• the current equipment inventory was used; 

• current prices for equivalent co:?^uters were used for all 
computer hardware; 

• the cost of hardware and software products was amortized over 
five years; 

• the cost of any infrastructure, like special furniture and 
cabling, was amortized over ten years; 

• the cost of any initial professional development for teachers 
was amortized over five years; and, 

• the cost of any new staff, staff development, and materials and 
supplies was treated as an annual expense. 

Three things stand out in these data. First, these schools all have 
lower student-to-computer ratios than the current national average of 
12:1, and in most instances, much lower ratios. Second, the two dominant 
factors contributing to total cost per student are computer hardware and 
cost of additional personnel needed to run the technology progreims. 
Third, the cost of software is small by comparison with the cost of 
hardware, unlike most enterprise computing^^, and small absolutely. 
Keltner spoke to this at the workshop 

The graph below shows the sensitivity of the cost per student to 
the student -to -computer ratio. 



The expectation in enterprise computing is that the cost of 
software will approach and equal the cost of hardware. 
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Fig* 1— StxxdAnt to CoapvLt%r Ratio 

The chart and table suggest a story about annual student cost. When 
the number of school computers is relatively modest, like one for every 
seven to eleven students, costs like technology-related staff, staff 
development, and materials and supplies will tend to dominate total 
cost; when the number of school computers increases to one for every two 
or three students, variable costs for hardware, software and 
infrastructure will tend to dominate. 

Software can include system, application, network and reference 
products; and application software can be further divided into tools, 
i.e., feature rich and content poor like word processing software; and 
content software, i.e., content rich and feature poor like common drill- 
and-practice software. Keltner and Ross speak to the relatively low 
level of school expenditures they found for software: ''Software 

expenditures account for 10% or less of total technology costs at all 

Drill-and-practice software is the most familiar current exaunple 
of content software, closely tied to the common scope and sequence 
curriculum in use by most schools. Content software need not in 
principle exclude products that aim to support a project-based 
curriculum. In practice, little content software of either kind exists 
for junior high and secondary school grades. 
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eight: schools. Software costs are on average one-guarter to one- fifth of 
total hardware costs. The school environment is not one that puts 
sophisticated demands on the software component of a technology program. 
The number of basic software programs installed on individual students’ 
computers is typically limited. Non of the schools in this survey 
purchased site licenses for more than 5-6 'tool-based' software 
products, e.g., Microsoft Word, Clarisworks, Hypertext or HyperCard, and 
the average was more like three. With a site license for 25 computers 
costing between $1000-$1500, an expenditure of $3000-$4000 typically 
pr'oved enough to outfit an entire classroom of computers with basic 
software applications . 

"Another explanation for the low level of software expenditures is 
the ability of schools to generate economies of scale in the use of 
expensive software products. The Christopher Columbus, Corona and 
Elizabeth St. schools each spent $30, 000-$40, 000 to set up large 
libraries of CD-ROM ani videodisc software products . CD-ROM and 
videodisc products are an important source of both 'content' and 
'reference' software. Blackstock and Taylorsville spent $43,00 and 
$70, 000 respectively on network and instructional management software. 
While expensive, these software items do not increase software 
expenditures per student significantly, because their cost is 
distributed over a large number of students . CD-ROM and videodisc 
products, are u. ed in the same way as library reference books — they are 
stored in one place for use by all students . Network and instructional 
management software are normally used on a school-wide or classroom- 
based LAN. " 

An earlier workshop^^ suggested an apparent market failure in 
content software, especially at the junior high and secondary school 
levels. The absence of products may be explained hy the low level of 
school expenditures. School expenditures for software in 1994 were less 
than $750 million, or an average of $7500 per school for the nation's 
100,000^^ or so schools. This figure compares with under $400 million in 

James Harvey (ed.). The Market for Educational Software, DRU- 
1041-CTI, RAND, Santa Monica, CA, May 1995. 

Used as a nominal figure throughout this summary. 
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consxjuner sales for CD-ROMs in the very first year that 'family' 
coit^uters came equipped with CD-ROM players. 

Whatever the cause, the shortfall in available content application 
software raises questions about whether students, on average, are 
realizing the full benefit of the school technology implementations, 
whatever the school's primary curriculum choice, linear scope and 
sequence, or many-patheu project-based; and the extent to which fully 
prepared teachers may be able to condensate for the shortfall in content 
applications . 

EFFECTZVBMESS 

Data on the benefits of optimal school-wide technology 
implementations, especially implementations in the service of school 
reform that aims at new student outcomes, new approaches for assessing 
student outcomes, and new instructional strategies (e.g., a significant 
measure of individualized student learning using a many-pathed, project - 
based curriculum) are not and will not soon be available. James Kulik, 
Bill Hadley, Dexter Fletcher and Luis Dsin provided different but 
overlapping slants on what we know from experimental and empirical data 
about the effectiveness of student learning using computer technology 
for the case of limited and well defined curriculum objectives. (We 
discuss later in the section titled 'evaluation' what can be inferred 
from this knowledge. ) 

James Kulik (University of Michigan) 

Kulik's presentation was based on his recent article, which 
opened with, "What do evaluation studies say about computer-based 
instruction? It is not easy to give a simple answer to the question. The 
term computer-based instruction has been applied to too many different 
programs, and the term evaluation has been used in too many different 
ways." He goes on to describe the meta-analytic approach for creating a 
composite picture of findings on computer-based instruction, and 

Meta-Analytic Studies of Findings on Computer-Based Instruction 
by James A. Kulik, 1994, in Technology Assessment in Education and 
Training, E.L. Baker and H.F. O'Neil, Jr., (eds.), Hillsdale, N J : 
Lawrence Erlbatm. 
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presents an overview of these findings. We quote: ''At least a dozen 
meta-analyses have been carried out to answer questions aUDOut the 
effectiveness of conputer-based instruction (Table 1.1, in original). 

The analyses were conducted independently by research teams at eight 
different research centers. The research teams focused on different uses 
of the coir^uter with different populations, and they also differed in 
the methods they used to find studies and analyze study results . 
Nonetheless, each of the analyses yielded the conclusion that programs 
of con^uter- based instruction have a positive record in the evaluation 
literature . 

"The following are some major points emerging from these meta- 
cuialyses : 



Students usually learn more in classes in which they receive 
computer-based instruction. The analyses produce slightly 
different estimates of the magnitude of the computer effect, 
but all the estimates were positive. At the low end of the 
estimates was an average effect size^^ of 0.22 in 18 studies 
conducted in elementary eind high school science courses 
(Willett, Yaroashita & Anderson, 1983) . At the other end of the 
scale, Schmidt, Weinstein, Niemiec, and Walbert (1985) found an 
average effect size 0.57 in 18 studies conducted in special 
education classes. The weighted average effect size in the 12 
meta-analyses was 0.35^®. This means the average effect of 
computer-based instruction was to raise examination scores by 



The meaning of effect size used here is the so-called 
standardized mean difference. This index gives the nimiber of standard 
deviation units that separates outcome scores of experimental and 
control groups. It is calculated by subtracting the average score of the 
control group from the average score of the experimental group and 
dividing the remainder by the standard deviation of the measure. T r;m 
Kulik, "For example, if a group that receives computer-based coaching on 
the Scholastic Aptitude Test (SAT) obtains an average score of 550 on 
the test, whereas a group that : eceives conventional teaching averages 
500, the effect size for the coaching treatment is 0.5, because the 
standard deviation on the SAT is 100. 

An effect size of 0.32 c= n be thought of as equivalent to a gain 
of about three months on an age-grade equivalent scale. 
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0.35 standard deviations, or from the 50th to the 64th 
percentile . 

• Students learn their lessons in less time with computer-based 
instruction. The average reduction in instructional time was 
34% in 17 studies of college instruction, and 24% in 15 studies 
of adult education (C.-L. C. Kulik & J. A. KuliiC, 1991). 

• Students also like their classes more when they receive 
computer help in them. The average effect of computer-based 
instruction in 22 studies was to raise attitude-toward- 
instruction scores by 0.28 standard deviations (C.-L. C. Kulik 
Sc J. A. Kulik, 1991) . 

"This brief review shows that there is a good deal of agreement 
among meta-analysts on the basic facts about computer-based instruction. 
All the meta-analyses that I have been able to locate show that adding 
computer-based instruction to a school program, on the average, improves 
the results of the program. But the meta-analyses differ somewhat on the 
size of the gains to be expected. We need to look more closely at the 
studies to determine which factors might cause variation in meta- 
analytic results." 

Kulik goes on to examine more closely a set of 97 evaluations of 
computer-based instruction in an attempt to reac more precise 
conclusions about their effectiveness, and concludes: "Meta-analysts 
have demonstrated repeatedly that programs of computer-based instruction 
usually have positive effects on student learning. This conclusion has 
emerged from too many separate meta-analyses to be considered 
controversial. Nonetheless, results are not the seime in every study of 
computer-based instruction. No meta-analyst has reported that all types 
of computer-based instruction increase student achievement in all types 
of settings. Study results are not that consistent, nor would we want 
them to be ^ Computer-based instruction is a loose category of 
innovation. It covers some practices that usually work and other 
programs that have little to offer. 

"Breaking studies of computer-based instruction into conventional 
categories clarifies the evaluation results. One kind of computer 
application that usually produces positive results in elementary and 
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high school classes is cortqputer tutoring. Students usually learn more 
in classes that include computer tutoring. On the other hand, precollege 

results are uninqpressive for several other conputer applications: 

20 . . • 
managing, simulations, enrichment, and programcaing . • 

"The overall findings on computer tutoring cortpare favorably with 
findings on other innovations. Few innovations in precollege teaching 
have effects as large as those of conputer tutorials. Effects are 
especially large and consistent in well designed programs such as the 
Stanford-CCC program. Programs of curricular change that provide more 
challenge, for high-aptitude students may have produced more dramatic 
effects in evaluation studies, but such programs affect only a limited 
part of the school population. The effects of conputer tutoring are as 
great as those of peer- and cross-age tutoring, and they are clearly 
greater than the gains produced by instructional technologies that rely 
on print materials.*' 

Bill Bmdley (ijaiiglay High School) 

Hadley, a teacher on a year's leave at Camegie-Mellon University 
(CMU) from Langley High School, Pittsburgh, PA, reported on the 
Pittsburgh Urban Mathematics Project (PUMP) . Based on a curriculum that 
emphasizes multiple representations (words, symbols, graphs, etc.), the 
program partially depends for its delivery on software - a so-called 
intelligent tutor — designed and developed by John R. Anderson, Prof, of 
Psychology and Conputer Science at CMU. PUMP has been operational at 
Langley H.S. for three years. 

With many of its students eschewing algebra in favor of a general 
mathematics track, which practically proscribes the necessary 
achievement in mathematics and science education for a successful 
technical career of any sort, Langley sought a solution in the 
combination of a revised algebra curriculum^^ and Anderson's algebra 

Kulik defines computer tutoring as a program in which the 
computer presents material, evaluates responses, determines what to 
present next, and keeps records of progress. Drill-and-practice software 
belongs in this category. 

As in computer-managed instruction. 

Developed primarily by Bill Hadley. 
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tutor, which was developed over a period of years with National 
Science Foundation R&D support. A statement of Langley's main objectives 
is ! 

• to have all students be successful in first year algebra and 
geometry; 

• to increase the number of students in higher mathematics 
classes ; 

• to have all students make conceptual and practical connections 
between algebra and the world outside of school; and, 

• to prepare students for the 'world of work' as well as further 
academic study in mathematics. 

Students spend five periods a week in algebra study, two with the 
intelligent tutor in a laboratory setting and three in a regular 
classroom. Carrying out exercises, which is greatly facilitated by the 
computer laboratory, is a primary element of an instructional strategy 
aimed at improving student learning. 

As a result of a satisfyingly successful experimental first year in 
which 73% of the students enrolled in the PUMP first-year algebra course 
passed, while 56% of students enrolled in the regular algebra course 
failed, the PUMP first-year algebra course was made a required course of 
study for all Langley students. In the second year of the adoption, 61 
of the 73 students passing the PUMP first-year algebra course enrolled 
in geometry; and of those 37 enrolled in Algebra 2 in year three. By 
contrast, 20 of the 24 students passing the regular algebra course 
during the experimental first year went on to take geometry; and of 
those only three enrolled in Algebra two. Two other schools in the 
Pittsburgh area recently adopted PUMP, despite the finemcial investment 
in hardware this step requires, but have not yet made first-year algebra 
a required course for all students. 

The computer equipment for a laboratory implementation of the 
program was donated by the Apple Computer, Inc. 
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J. D«act#r Flmtdhmr (Ixistituta for Dofonso Axuilysos) 

Fletcher, speaking to the use of technology in military training, 
started by remarking on a number of features distinguishing military 
training from K-12 education: 



• Military training involves bringing individuals or collections 
of individuals to a required level of performance in the 
conduct of prescribed tasks. The time to reach this level of 
coiipetence, a variable among trainees, affects training cost in 
several ways, and therefore cost-effectiveness, which sharply 
affects the selected training approach. 

• The costs of training (most of which are time dependent) that 
employs 'hard' technology can be roughly divided into three 
categories: the trainee cost, the hard technology cost, and an 
overhead cost. The trainee is paid while in training; there is 
an added cost if the trainee is removed from the field and a 
replacement is required; and travel and living expenses for the 
trainee and any replacement may constitute yet another cost. 

The costs for hard technology, say a flight simulator, include 
R&D, and production and maintenance, which have to amortized 
over the lifetime of the equipment, and contribute a time- 
dependent amount to the cost of the individual's training. 
Overhead costs include everything else, like maintenance of the 
training site, and so on, which also contribute a time- 
dependent amount to the total cost of the individual's 
training. Time to train is therefore a major consideration for 
DoD. 

• Unlike the case for schools, the locus of decision-making for 
the use of technology in military training is not local and 
decentralized, but centralized and even highly centralized 
where initial R&D support is required. 

• In suiranary, the DoD uses technology to regain the benefit of 
individual tutoring that has been lor t to the economic 
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necessity of training students in large classes. Bloom^' writes 
that the difference between individualized tutoring and group 
instruction may account for as much as two standard deviations 
in measured achievement. 
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Effect;iv«&MS. Fletcher then went on to discuss the issues of 
effectiveness, cost and cost-effectiveness in turn, pointing out that 
the available data for considering these training criteria diminish in 
that order. Emphasizing the importance of individualization (for pace, 
difficulty, content, secfuence and style) to effectiveness, he presented 
data on effect sizes for training using interactive (computer- 
controlled) videodisc instruction coitqpared with more conventional 
approaches like platform lecture, textbooks, workbooks, and the use of 
actual equipment for practice. Videodisc functionalities in these 
comparisons give some indication of the effectiveness of multimedia 
approaches to instruction. The average effect size across 47 evaluations 
of interactive videodisc instruction used in military training, 
industrial training, and higher education was 0.50, or an increase from 
50th to 69th percentile levels of performance. Considering the three 
settings separately, the effect size was 0.39 for military training (an 
increase from 50th to 65th percentile performance), 0.51 for industrial 
training (an increase from 50th to 70th percentile performance) , and 
0.69 for higher education (an increase from 50th to 75th percentile 
performance) . 

Little difference in effectiveness is found between knowledge and 
skill performance measures in interactive videodisc instruction 
evaluations; i.e., interactive videodisc instruction appeared to be 
equally effective for both. Studies that compared the interactive 
intensity of interactive videodisc instruction found signif iceintly 
better results for the more interactively intense applications. 

The average effect size in 38 studies of military training using 
computer-based instruction (CBI) is reported as 0.40 for experimental 

B. S. Bloom, "The Two Sigma Problem: The Search for Methods of 
Group Instruction as Effective as One-to-One Tutoring," Educational 
Researcher, 13, 1984, pp. 4-16. 
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groups numbering less than 20, in the same range as effect size in 
education reported by Kulik; cind 0.30 averaged over 35 studies for 
experimental groups numbering greater than 20. 

Effect size has also been used to assess simulation applications* 
for training to maintain and repair devices, that is, the use of 
simulated equipment is compared to the use of actual equipment, with 
training time held constemt and success in maintaining or repairing 
actual ecjuipment 'used as the final performance measure. Average effect 
size in these studies has been found to be 0.40 (an increase from 50th 
to 66th percentile performance) . And most notably, the cost of training 
using simulated equipment was found to be about a third the cost of 
training using actual ecjuipment. 

Evaluations of interactive videodisc used to simulate actual 
equipment have been performed using two approaches: videodisc used only 
in simulator mode and videodisc used in both simulation mode and for 
tutorial guidance during the simulation. Effect sizes for these two 
approaches were 0.14 (an increase from 50th to 56th percentile 
performance) and 0.41 {an increase from 50th to 66th percentile 
performance), respectively. This finding is consistent with others that 
have found an interaction effect between the sophistication of the 
students and the amount of tutorial guidance needed in simulation-based 
training. 'Naive' students benefit from tutorial assistance. 

Finally, Fletcher reported on a. group of assessments of computer- 
based instruction in K-12 education, which predicted performance on 
standardized achievement tests solely on the basis of the amount of time 
each student spent to complete a preparatory program of computer-based 
instruction. Scores on the comprehensive tests of mathematics 
achievement could be predicted to the nearest tenth of a grade-placement 
using these time measures exclusively. Faster learners are better 
learners. 

Cost. In military training, the time required to train to a 
required level of performance sharply affects almost all cost elements 
contributing to total cost. Repeated analyses have found that, on 
average, technology reduces the time to reach criterion levels of 
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knowledge and performance by about 30 percent, in the same range of 
reduced instru.ctional time in education reported by Kulik. 

Cost Effsctivsnsss. For the case of military training by 
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simulation, which can never entirely replace the 'real thing' , 
assessing the cost-effectiveness of simulation, whether- by general- 
purpose hardware or special ec[uipment, (and setting aside any other 
costs,) requires consideration of an additional factor, the transfer 
effectiveness ratio (TER) . For a flight simulator, for example, this 
would be computed as the difference between actual aircraft time without 
simulator training and aircraft time with simulator training (each of 
which has a cost) divided by simulator time. 

That is, up to the limit for which the simulator can realistically 
simulate general air work, an hour of simulator time saves an hour of 
training time in an actual aircaft. For this case, if the cost of an 
hour of simulator training is less than the cost of an aircraft hour, 
simulator time is cost-ef fctive . 

For K-12 education, Fletcher summarized his paper^^ comparing the 
costs (in constant 1985 dollars) to increase comprehensive mathematics 
scores (computation, concepts and word problems) one standard deviation 
using different approaches: tutors, reduced class size, increased 

instructional time, and providing computer based instruction. The 
results appear below. 



K-12 education does not much consider this, which would give new 
meaning to the stated goal of preparing students for the world of work. 

D. Fletcher, D. E. Hawley, and P. K. Piele, "Costs, Effects 
and Utility of Microcomputer Assisted Instruction in the Classroom," 
American Educational Research Journal, 27, 1990, pp. 783-806. 
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Approach 



Cost for 1 sd gain ($) 



Tutoring (20 min /day) 



by peers 286. 

by adults 1612. 

Reduced class size 

35 to 30 983. 

30 to 25 1171. 

25 to 20 1367. 

35 to 20 1195. 



Increased instructional time 
30 min /day 2667. 



CBI (10 inin/day 3rd grade and 11/2 min/day 5th grade)) 



grade 3 (coirqputation) 338. 
grade 3 (concepts) 208. 
grade 3 (word problems) 192. 
grade 5 (computation) 462. 
grade 5 (concepts) 490. 
grade 5 (word problems) 206. 



Peer tutoring and CBI are revealed by these data to be the most 
cost-effective approaches. Fletcher observed these two approached could 
be combined and reconciled by having two or three students clustered 
together at a single CBI terminal . 

Slimming up the military experience, Fletcher offered the following 
principles to guide the use of technology for training: 

• for practice rather than initial learning; 

• to simulate expensive equipment or dangerous field conditions; 

• provide self-study for remote or dispersed learners; and, 

• to closely monitor progress in student learning. 

Concerning the conceptualization and practice of assessment, he 
suggested assessments should: 



be designed to inform specific choices; 

address both formative (design issues) and summative issues; 
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• expand the range of instructional outcomes considered and 
support the development of principles of instructional design 
that eitphasize specifiable outcomes; 

• validate instructional objectives to assure not only improved 
instruction, but improved instructional objectives; 

• consider broad ranges of instructional inputs and outcomes, 
rather than narrow ones; 

• consider group performance; and, 

• consider cost. 

Luis Osin (Cantre for Educational Tachnology, Xsraal) 

Luis Osin, on leave at the Learning Research & Development Center 
of the Univ. of Pittsburgh from the Centre for Educational Technology 
(CET) in Israel, reported on the GET computers in instruction program in 
Israel, where almost 700 schools use CET coit^uter systems comprising 
some 18,000 student stations utilized by nearly 180,000 students 
annually. With the goal of adapting instruction to the individual 
learner ~ the learner's current knowledge, cognitive learning style and 
pace, CET offers a full range of services, including advice to schools 
and education authorities, supply and installation of computerized 
systems, in-school teacher training, and full system maintenance of 
hardware, software and coursew?^xe (application software) . Indeed, almost 
30 communities with some 150 schools have elected to operate under the 
direct supervision of CET, while hundreds of additional schools have 
adopted some of its adaptive teaching and learning methods. 

Starting with the observation that individualized instruction is 
necessary to overcome the well-known distribution of the age-grade of 
students at any grade level, (e.g., nominally 4th grade students 
typically range in achievement between 2nd and 6th grades; nominally 6th 
grade students typically range in achievement between the 3rd and 9th 
grades; and so on,) Osin went on to describe the role of conputers in 
CET's adaptive teaching and learning strategy: 

• Individualized dialog with every student . By interacting with 
the computer, every student may learn according to his/her 
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cognitive level and learning speed, independently of the 
cognitive styles and learning pace of the student's classmates. 
The student is able to hold a 'conversation' with the 
(software) author, and receive explanations matched to his/her 
level of learning . 

• Tools for information p rocessing . Teachers and students may* 
enjoy using general purpose (feature rich) tools like text and 
93^sphic editors, spreadsheets, etc. The teaching and learning 
of many subjects can be based, at least partially, on the 
utilization of these tools. 

• Access to remote information. Today, it is possible to access 
large and updated data b a n ks , located not only in one ' s own 
country, but dispersed around the world. 

• Communication with ot hers around the world . Students in 
different schools, cities and countries can cooperate on a 
common project. Teachers can benefit from advice and support of 
remote educational R&D centers. 

• £. timulating presentation . Material can be presented to the 
student with all of the expressive possibilities of modem 
c inema togr aphy . 

A well-rationalized program includes intensive teacher preparation 
and the engineering design of content software (i.e., many cycles of 
trial and improvement) , which may be more top-down than would be 
acceptable in U.S. schools. The result was revealed in one exait^le of a 
school with a low SES population in which an exceptionally and unusually 
high percentage of the entire student body was performing on or above 
grade level in mathematics achievement. 

Osin observed that the principal beneficiaries of individualized 
instruction were slow or low-aptitude learners, and that computer 
resources could be allocated in such a way as to prevent them from 
falling behind their age cohort, with the usual educationally 
destructive consequences. 
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implementation issues and strategies 

For schools, the problems of implementing a technology based 
approach to improve student learning include teacher buy-in and 
preparation, a reorganization of the school day to allow time for 
student use,^^ and financing the cost. Focusing the technology 
implementation on individualization of instruction introduces issues of 
computer curriculxim (i.e., courseware or content software), and 
assessment and monitoring of individual student progress towards the 
achievement of consensual learning goals. Refocusing the inplementation 
on the regnant definition of school reform introduces the additional 
issues of an instructional strategy defined by a project approach to 
student learning and a portfolio approach to student assessment. Barbara 
Means, Martin Huntley and David Dwyer spoke to individual school 
implementation problems and strategies . 

Barbara Maans (SRI^ 

Means reviewed a U.S. Department of Education (Office of 
Educational Research and Improvement) sponsored study of social and 
organizational factors affecting technology and school reform. The 
schools had various motivations for their bottom-up efforts to adopt 
technology, including: 

• the belief that use of computer-based technologies could 
support the development of the student's thinking processes; 

• the belief that use of computer-based technologies could 
stimulate student motivation for learning, and tend to elevate 
self-esteem for low-SES students especially; 

• promote equity for students less likely to experience computer 
technology at home; 

• prepare students for the world of work; and, 

H. J. Becker, Analysis and Trends of School Use of New 
Inforrnation Technologies , prepared for the Office of Technology 
Assessment of the U.S. Congress, October 1993. 

Details of nine case studies of schools, with high concentrations 
of students from low-income homes, using technology to support their 
education reform efforts may be found at http://www.ai.sri.com/edtech/. 
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• support changes in school structure. 

Means reviewed the implementation story for two of the nine 
schools, both middle schools, one argruably a success in its effort and 
the other a patent failure, for the lessons these different outcomes 
offered. The technology adoption goal was initially conceived at the 
school district level in both cases; both schools expended significant 
new resources on the implementation ; both schools were new, opening in 
1991 and 1989 respectively; and both schools served the children of 
mostly low- income families. 

The approach towards the goal of technology adoption of one school 
was characterized by: 

• careful planning, with a principal hired for the express 
purpose one year in advance; ■ 

• technology manager hired to manage the school network; 

• a school schedule organized to give teachers 90 minutes a day 
for planning and collaboration; 

• building renovation designed to support interdisciplinary 
teacher teams with common students and shared office space; 

• technology distributed throughout regular classrooms; and, 

• extensive teacher training provided before the school opening, 
and continuing regularly with faculty meetings and in-service 
training days; 

with something like the following observed outcomes: 

• the majority of teachers use technology for instruction; 

• students and teachers take pride in their school; 

• students value their teachers first, technology second; 

• low teacher turnover; 

• students have technology skills; 

• students perform better than their counterparts in other 
schools on State assessments. 
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Perhaps by contrast, the approach of the second school towards the 
goal of technology adoption was characterized by: 

• extensive planning at the school district level, with hand- 
picked lead teacher responsible for development of curriculum 
framework hired just before school opening; 

• aim to individualize student educational program based on 
individual interests ; 

• a system for tracking student learning goals and 
accomplishments ; 

• innovative staffing concept with lead and associate lead 
teachers responsible for curriculum framework and curriculum 
design, and general classroom teachers added as newly opened 
school added grades ; 

• use of outside resources like museums and art galleries on a 
regular basis; and, 

• specially designed spaces to support specific kinds of learning 
and instruction; 

with something like the following observed outcomes: 

• by five years after opening and initiation of technology 
adoption goal, excitement dissipated and period of very high 
staff turnover began; 

• some hardware was dispersed to other schools, and much of the 
remainder lay idle on most days; 

• many teachers did not use technology in their teaching; and, 

• student test scores were lower than expected, and school 
acquired reputation as a 'dumping ground'. 

In explanation of this unsatisfactory result. Means offered the 
following : 

• a mismatch between district-selected technology and the staff's 
instructional philosophy; 
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• planned investment in training teachers was never carried out; 

• schism developed between lead teachers responsible for 
curriculum framework and teachers responsible for classroom 
instruction; 

• part-time technology coordinator had little contact with 
classroom teachers; and, 

• publicity turned sharply negative after siTiall, initial drop in 
student test scores . 

^^rawing on the nine case studies, Means ventured the following 
features required for a successful technology adoption: 

• jointly developed school goals> and technology's place in 
fulfilling them; 

• adequate technology access provided in regular classes; 

• technical support readily available and non- judgmental; 

• professional growth opportunities, recognition, and rewards 
provided for exemplary technology use by regular teachers; 

• technology use as a choice, not by fiat; 

• mechanisms for teacher choice in what technology to use and how 
to use it; 

• opportunities provided for teachers to work together; and, 

• supported time for teachers to leam to use technology and to 
design technology-supported learning activities . 

Martin Huntley (BBM) 

Huntley briefly described pha.<^e 2 of a National Science Foundation 
sponsored testbed project aimed at assisting schools "to build the local 
information infrastructure (LII) in such a way that all participants in 
schools and local communities can actively construct networked services 
affecting all aspects of learning, teaching, administration and 
community access to education." Designed as a grass-roots effort, the 
project has attracted the participation of 95 member organizations, 
representing some 200 schools with Internet connectivity in 39 States; 
some state education agencies, regional education labs, museums, 
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educational R&D organizations and universities; and some commercial 
firms like school publishers, service providers and system vendors. 

Three key project assumptions are that: 

• the school and community LII needs to be locally constructed to 
support the community educational mission; 

• the construction must go hand in hand with creating a learning 
environment, which supports and promotes achievement with 
respect to new and emerging standards; and, 

• the testbed activities will provide an environment for 
collaboration in addressing the daunting and complex array of 
issues faced by project members. 

Year one project priorities are: 

• teacher development: building a technology culture within the 
teaching community; 

• funding, financing: helping communities develop strategies to 
foot the bill; 

• collaboration mechanisms: tools and strategies for on-line 
collaboration for use within the testbed community and by 
members within their LIIs; and, 

• evaluation: helping members tell whether their technology 
initiatives are making a difference. 

David Dwyer (Apple Coonputer) 

Dwyer reviewed the Apple Classroom of Tomorrow (ACOT) project, 
initiated in 1986 in seven classrooms representing a cross section of 
U.S. education. Providing two computers (one for school and one for home 
use) to each student and teacher, and ample teacher support in a project 
that depended principally on teacher initiative, ACOT also initiated a 
long-term, university-based project to track developments and classroom 
change. Two years into the activity, findings included: 

• Teachers were not hopeless technical illiterates. 
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• Students did not become social isolates. 

• Student's interest in and engagement with the technology did 
not decline with routine use. 

• Students, even young children, did not find the keyboard a 
barrier to fluid use of the cortqputer. 

• Software did not prove to be a limiting factor, even in high 
school classrooms where Macintosh conputers were the machines 
of choice, and hardly any educational software existed. 

• Attendance was up, and test scores were generally un.changed. At 
one site, where coirqputers were purposefully used to raise 

. student test scores, ACOT students outscores non-ACOT students 
significantly on the California Achievement Test in vocabulary, 
reading comprehension, language mechanics, math computation, 
and math concept /application. 

• /\COT students wrote more, more effectively and more easily than 
non-ACOT students . 

In the third and fourth years of the activity, as teacher comfort 
with the eguipment was established, (some) teachers began experimenting 
with new tasks for students and most K-6 teachers had successfully 
modified daily schedules to permit students more time to work on 
coiiputer-based projects. Teachers struggled with the issue of methods of 
assessment that would capture the novel ways in which students were 
demonstrating their mastery of skills and concepts. 

Dwyer siammarized the catalytic impact of technology in stimulating 
educational improvement and classroom change: 

• Technology encourages fundamentally different forms of 
interaction among students and between students and teachers. 

• Technology engages students systematically in higher-order 
cognitive tasks ; and, 

• Technology prompts teachers to question old assumptions about 
instruction and learning. 
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He concluded by remarking that successful classroom change 
catc^lyzed by the computer depended upon new methods of assessment of 
student and teacher performance, and required broad support by 
administrators in the school and school system in their role as 
instructional leaders . 

EVJOiUATlON 

Evaluating technology effectiveness in education poses special 
problems, because of additional educational changes, beyond the 
technology, needed to reveal the technology's full promise. Douglas 
Merrill of Rand spoke to this problem. Larry Prase volunteered that in 
the circumstance of rapid technological change, the much slower pace of 
educational software development, and the relatively glacial rate of 
change in new knowledge from research on software usability and on 
models of implementation in different subject inatters, an engineering 
approach to evaluation might be preferable to the idealized scientific 
model of controlled experimentation. He subsequently wrote on the 
subject from which we quote below. 

Douglas Hsirrill (RAND) 

Merrill emphasized that the opportunity of using technology to 
improve student learning and school effectiveness could readily be 
thwarted by an evaluation approach that was collinear with existing 
educational practice and ignored factors affecting an effective 
technology implementation like: 

• pedagogy, e.g., more student- focused than teacher-centered; 
and, 

• time and curriculum framework, e.g., longer, multidisciplinary 
classes. 

He went on to say that the metrics employed in an appropriate 
evaluation should focus on the interaction of student, institutional and 
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task factors. From a paper by Merrill^®: Naturally, the student's 

personal characteristics interact with the educational intervention to 
produce differing outcomes. For example, there are students who exhibit 
higher abili y than others, regardless of one's beliefs about the nature 
of the vague construct 'ability' . Higher ability students tend to react 
differently to feedback than students who are having more difficulty. 

. . . differences in feedback behavior are often directed at maintaining 
students' motivation, another key personal' characteristic affecting 
pedagogical outcomes. Thus one would expect ability — however defined - 
and motivation to be important factors to consider when evaluating 
educational technology'. 

Concerning instit-utional factors, Merrill writes: Student factors 

are not all that should affect the metrics used by an evaluation. The 
way the technology is integrated within the school setting also can 
change the set of appropriate measures. For example, it is often the 
case that schools do not place conputer-based tools in many classrooms, 
but rather centralize them in a computer laboratory, run by a set of 
'high priests'. 

This design does not lend itself to a tight integration of 
technology into curriculum, because using the machines requires 
reserving the laboratory, getting the students there, . . . For excjnple, 
one common early measure of the effect of technology on education was 
the total time spent using the computers per class. Given a laboratory 
design, this particular metric is virtually guaranteed to be small — but 
interpreting this small amount of time on computer tasks as indicative 
of a failure of technology is probably short-sighted. 

Again, Thus, another possible source of interference with 
evaluation of educational technology is variance in the training of 
teachers, and the consequent differences in comfort level and curricular 
integration. 

On task factors, Merrill writes: The nature of the task put to 
students and the knowledge they are expected to acquire should also 

“ Douglas C. Merrill, "Evaluation of Educational Technology: What 

Do We Know, and What Can We Know, * institute on Education and Training, 
RAND, Santa Monica, CA, March 1995. 
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affect the evaluation metrics selected and their interpretation. A task 
could he primarily procedural in nature, requiring the application of a 
sequence of steps, hut not necessarily drawing on many facts, known as 
declarative knowledge. Much of mathematics consists of procedural tasks, 
with the declarative portion devoted solely to selecting a procedure to 
apply. In contrast, the study of history involves the recognition of 
relationships between actors and events declarative knowledge. Again, 
an evaluator must determine which of these two types of knowledge to 
measure and to interpret the results accordingly. 

Finally, Future evaluations of educational technology should not 
focus single-mindedly on simple outcome measures, such as post-tests, 
without also attending to more complex metrics describing the learning 
process, such as cognitive modeling. Furthermore, the research must also 
take the institutional factors into account, such as where a changed 
curriculum fits and what other changes must ^ccoirpany its introduction. 
Similarly, each different task proposed as part of a new curriculum 
could require different sorts of evaluations. 

Larry Frase ( ETS ) 

Frase wrote later suininarizing his thoughts about the workshop. We 
quote on the issue of evaluation. 

"The workshop showed that we can expect increasing demands for 
teacher training, partnering to achieve the best and most efficient 
educational implementations, demands for new research and measurement 
techniques and for more formal approaches to educational implementation, 
and demands for the development of broadly applicable telecommxinications 
and educational software tools. 

The technologically derivative revolutions in education and 
research pose a peculiar problem, because of the temporal 
discontinuities associated with technological change. Consider the time 
scales within which progress takes place in technology, education, and 
research and what these discontinuities portend. The rapid advance of 
computing sneed and miniaturization is well known. The development of 
software, to make use of those capabilities, appears to lag, but 
research on software usability and models for educational implementation 
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in different subject matters lags even more. Developments in educational 
software, in fact, are impeded by archaic models of educational theory. 

*The idealized model of scientific research, as controlled 
experimentation leading to the resolution of major theoretical issues, 
is probably the wrong model for education in an age of rapid technology 
development. A more appropriate model would be an engineering approach, 
using the tools and techniques of science to evaluate evolutionary 
(formative) changes in educational products and processes. Companies, 
such as AT&T and IBM, have already pushed their research communities in 
this direction. It is only a matter of time before the academic 
community does the same. 

"The concept of educational achievement, as represented by the 
measurement of outcomes, must change. The outcome evaluations reported 
in this workshop were all the more impressive, because they represent 
weak tests — many of the evaluative instruments were quite general, only 
weakly focused on the concepts taught and quite probably containing 
irrelevant elements. In short, the concept of educational achievement, 
as represented by global outcome tests, lacks conceptual validity. We 
know that educational outcomes consist of more than what we can see on a 
few test items. If we can better define the processes involved in 
thinking in different subject matter domains, more reasonable 
instruction and assessment will follow. To the extent that this is true, 
the data on the effects of educational technology presented in this 
workshop were an answer to the wrong question. In cases where the 
elements of instruction were developed concurrently with tests of 
achievement, very strong educational effects were seen.* 
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APPBHDIX 1 

Workshop on Technology-Supported Student Learning: 
Outcomes, Effectiveness and Cost 

Rand’s Wash., D.C. Office Tel : 202-296-5000, x5321 

2100 M St., NW - 8th floor Fax : 202-2%-7960 

Chair: Tom Glennan Rapporteur : Jim Harvey 

Co-chair: Arthur Melmed Secretary : Nancy Rizor 



Thu 1 June 



8:30 AM - 9:00 AM 


Continental breakfast 




9:00 AM - Noon 


Opening remarks 


Linda Roberts, U.S. Dept, of Education 
Ed Fitzsimmons, White House O.S.T.P. 




Three school presentations 
with discussion (50 min. 
each) 


Steve Carr, Blackstock JrHS 
Fred Carrig & Bob Fazio, CC MS 
John Gibson, East Bakersfield HS 


Noon - 1:00 PM 


Lunch provided 




1:00 PM -5:00 PM 


Three school presentations 
with discussion (50 min. 
each) 


John Hadley, Langley HS 
Susan Wolf, Northbrook MS, 
Beth Stroh, Taylorsville, ES 


Fri 2 June 


Martin Huntley, BBN 


School technology implementation 
strategy 


8:00 AM - 8:30 AM 


Continental breakfast 




8:30 AM- 11:30 AM 


Douglas Merrill, Rand 
James Kulik, U. Michigan 
Dexter Fletcher, IDA 
Luis Osin, CET 


What we can know; and how (25 min.) 
What the education data say (50 min.) 
What the training data say (50 min.) 
What Israeli school data say (50 min.) 


ll:30AM-+2t30PM 


Lunch provided 




12:30 PM -4:00 PM 


Barbara Means, SRI 
David Dwyer, Apple 
Computer 


Social and organizational supports (50 
min.) 

The ACOT experience (50 min.) 




Group discussion 


What do we think we know; what more 
should we try to learn; and how. 
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APPENDIX 2 

XiZST OF WQRXSBOP PARTICIPANTS 



Linda Roberts, Director 

Office of Educational Technology 

U.S. Dept, of Education 

Wash., D.C. 20202 

tel : 202- 401-1444 

fax : 202- 401-3093 

Ed Fitzsimmons 

White House Office of Science and Technology Policy 
Wash., D.C. 20500 
tel : 202-456-6038 
fax : 202-456-6023 

Steve Carr 
Blackstock JrHS 
8042 Ventura St. 

Ventura, CA 93004 
tel : 805-488-0295 
fax : 805-488-1250 

Fred Carrig, District Director for Academic Programs 

Board of Education 

Union City School District 

3912 Bergen Turnpike 

Union City, NJ 07087 

tel : 201-348-2083 

fax : 201-348-5866 

Bob Fazio, Principal 

Christopher Coliimbus Middle School 

1500 New York Avenue 

Union City, NJ 07087 

tel : 201-271-2083 

fax : 201-271-2087 

John Gibson, Principal 
East Bakersfield High 
2200 Quincy St. 

Bakersfield, CA 93306 
tel : 805-871-7221 

fax : 805-872-6980 

Bill Hadley 
Langley High School 
1308 Milton St. 

Pittsburgh, PA 
tel : 412-268-3409 

fax : 412-268-2844 
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Susan Wolf, Principal 
Northbrook Middle School 
3030 Rosefield Ave. 

Houston, TX 77080 
tel : 713-329-6510 

fax : 713-329-6537 

Barbara Means 
SRI International 
333 Ravenswood Avenue 
Menlo Park, CA 94025 
tel : 415-859-40004 

fax : 415-859-2861 

Beth Stroh 

Taylorsville Elementary School 

PO Box 277 

9711 Walnut Street 

Taylorsville, Indiana 47280 

tel : 812-526-5448 

fax : 812-526-2233 

Martin Huntley 
BBN 

150 Cambridgepark Dr. 

Cambridge, MA 02140 
tel : 617-873-2850 

fax : 617-873-2455 

Douglas Merrill 
Rand 

1700 Main Street 
PO Box 2138 

Santa Monica, CA 90407-2138 
tel : 310-393-0411, x7876 
fax : 310-393-4818 

James A. Kulik 

Center for Research on Learning and Teaching 

109 E. Madison Street 

University of Michigan 

Ann Arbor, MI 48109 

tel : 313-936-0636 

fax : 313-936-0643 

Dexter Fletcher 

Science and Technology Division 
Institute for Defense Analyses 
1801 N. Beauregard St. 

Alexandria, VA 22231 
tel : 703-578-2837 

fax : 703-931-7792 
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Luis Osin 

LRDC [visiting address] 

Univ. of Pittsburgh 
3939 O'Hara St. - Riti. 516 
Pittsburgh, PA 15260 
tel : 412-624-8547 

fax : 412-624-9149 

Centre for Educational Technology [pennanent address] 
16 Klausner Street 
Raroat Aviv 61394 
ISRAEL 

fax : 972-3-642-2619 

David Dwyer 

Apple Computer 

1 Infinity Loop, MS:301-3e 

Cupertino, CA 95014 

tel : 408-974-4574 

fax : 408-862-6430 

Lawrence T. Prase 

Div. of Cog. and Instr. Sci. 

ETS 

Mail Drop 13R 
Princeton, NJ 08541 
tel : 609-734-1153 

fax : 609-734-1090 

Tom Glennan 
Rand 

2100 M St . , NW 

Wash., D.C. 20037 

tel : 202-296-5000, x5380 

fax : 202-296-7960 

Arthur Melmed 

The Institute of Public Policy 
George Mason Dniv. 

Fairfax, VA 22030 
tel : 202-244-9056 

fax : 202-296-7960 

James Harvey 

James Harvey & Assoc. 

1129 20th St . , NW 
Wash., D.C. 20036 
tel : 202- 659-4670 

fax : 202- 466-4040 
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